{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Basic Autoencoders\n",
    "- Objective: understand basics of autoencoders\n",
    "- References\n",
    "    - https://blog.keras.io/building-autoencoders-in-keras.html\n",
    "    - https://medium.com/@curiousily/credit-card-fraud-detection-using-autoencoders-in-keras-tensorflow-for-hackers-part-vii-20e0c85301bd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Autoencoders\n",
    "- One of the unsupervised neural network structures to obtain **compressed encodings**\n",
    "- Mainly used for dimensionality reduction, generative models, denoising etc.\n",
    "\n",
    "<br>\n",
    "<img src=\"https://upload.wikimedia.org/wikipedia/commons/2/28/Autoencoder_structure.png\" style=\"width: 600px\"/>\n",
    "</br>\n",
    "\n",
    "- Two main components of autoencoder; encoder and decoder\n",
    "    - Encoder compresses input into small set of \"codes\" (generally, dimensionality of encoder output is much smaller than encoder input)\n",
    "    - Decoder then expands the encoder output into output that has same dimensionality as encoder input\n",
    "    - In other words, autoencoder aims to \"reconstruct\" the input while learning finite representation of the data (i.e., \"codes\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load dataset\n",
    "- Digits dataset in sklearn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import normalize\n",
    "from sklearn import datasets\n",
    "from IPython.display import SVG\n",
    "from keras.utils.vis_utils import model_to_dot\n",
    "from keras.layers import Input, Dense\n",
    "from keras.models import Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "data = datasets.load_digits()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_data = data.images\n",
    "y_data = data.target"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_data = X_data.reshape(X_data.shape[0], 64)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# fit in data instances into interval [0,1]\n",
    "X_data = X_data / 16."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.    ,  0.    ,  0.3125,  0.8125,  0.5625,  0.0625,  0.    ,\n",
       "        0.    ,  0.    ,  0.    ,  0.8125,  0.9375,  0.625 ,  0.9375,\n",
       "        0.3125,  0.    ,  0.    ,  0.1875,  0.9375,  0.125 ,  0.    ,\n",
       "        0.6875,  0.5   ,  0.    ,  0.    ,  0.25  ,  0.75  ,  0.    ,\n",
       "        0.    ,  0.5   ,  0.5   ,  0.    ,  0.    ,  0.3125,  0.5   ,\n",
       "        0.    ,  0.    ,  0.5625,  0.5   ,  0.    ,  0.    ,  0.25  ,\n",
       "        0.6875,  0.    ,  0.0625,  0.75  ,  0.4375,  0.    ,  0.    ,\n",
       "        0.125 ,  0.875 ,  0.3125,  0.625 ,  0.75  ,  0.    ,  0.    ,\n",
       "        0.    ,  0.    ,  0.375 ,  0.8125,  0.625 ,  0.    ,  0.    ,  0.    ])"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_data[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X_data, y_data, test_size = 0.3, random_state = 777)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1257, 64)\n",
      "(540, 64)\n",
      "(1257,)\n",
      "(540,)\n"
     ]
    }
   ],
   "source": [
    "print(X_train.shape)\n",
    "print(X_test.shape)\n",
    "print(y_train.shape)\n",
    "print(y_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Simple autoencoder\n",
    "- Simple autoencoder with only one hidden layer\n",
    "- Hence, both encoder and decoder are composed of two layers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# define coding dimension\n",
    "code_dim = 6"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "inputs = Input(shape = (X_train.shape[1],), name = 'input')                         # input layer\n",
    "code = Dense(code_dim, activation = 'relu', name = 'code')(inputs)                  # hidden layer => represents \"codes\"\n",
    "outputs = Dense(X_train.shape[1], activation = 'softmax', name = 'output')(code)    # output layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input (InputLayer)           (None, 64)                0         \n",
      "_________________________________________________________________\n",
      "code (Dense)                 (None, 6)                 390       \n",
      "_________________________________________________________________\n",
      "output (Dense)               (None, 64)                448       \n",
      "=================================================================\n",
      "Total params: 838\n",
      "Trainable params: 838\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "auto_encoder = Model(inputs = inputs, outputs = outputs)\n",
    "auto_encoder.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<svg height=\"191pt\" viewBox=\"0.00 0.00 120.00 191.00\" width=\"120pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g class=\"graph\" id=\"graph0\" transform=\"scale(1 1) rotate(0) translate(4 187)\">\n",
       "<title>G</title>\n",
       "<polygon fill=\"white\" points=\"-4,4 -4,-187 116,-187 116,4 -4,4\" stroke=\"none\"/>\n",
       "<!-- 2108469287232 -->\n",
       "<g class=\"node\" id=\"node1\"><title>2108469287232</title>\n",
       "<polygon fill=\"none\" points=\"0,-146.5 0,-182.5 112,-182.5 112,-146.5 0,-146.5\" stroke=\"black\"/>\n",
       "<text font-family=\"Times New Roman,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"56\" y=\"-160.8\">input: InputLayer</text>\n",
       "</g>\n",
       "<!-- 2108469288800 -->\n",
       "<g class=\"node\" id=\"node2\"><title>2108469288800</title>\n",
       "<polygon fill=\"none\" points=\"13,-73.5 13,-109.5 99,-109.5 99,-73.5 13,-73.5\" stroke=\"black\"/>\n",
       "<text font-family=\"Times New Roman,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"56\" y=\"-87.8\">code: Dense</text>\n",
       "</g>\n",
       "<!-- 2108469287232&#45;&gt;2108469288800 -->\n",
       "<g class=\"edge\" id=\"edge1\"><title>2108469287232-&gt;2108469288800</title>\n",
       "<path d=\"M56,-146.313C56,-138.289 56,-128.547 56,-119.569\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"59.5001,-119.529 56,-109.529 52.5001,-119.529 59.5001,-119.529\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 2108469286728 -->\n",
       "<g class=\"node\" id=\"node3\"><title>2108469286728</title>\n",
       "<polygon fill=\"none\" points=\"9,-0.5 9,-36.5 103,-36.5 103,-0.5 9,-0.5\" stroke=\"black\"/>\n",
       "<text font-family=\"Times New Roman,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"56\" y=\"-14.8\">output: Dense</text>\n",
       "</g>\n",
       "<!-- 2108469288800&#45;&gt;2108469286728 -->\n",
       "<g class=\"edge\" id=\"edge2\"><title>2108469288800-&gt;2108469286728</title>\n",
       "<path d=\"M56,-73.3129C56,-65.2895 56,-55.5475 56,-46.5691\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"59.5001,-46.5288 56,-36.5288 52.5001,-46.5289 59.5001,-46.5288\" stroke=\"black\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "SVG(model_to_dot(auto_encoder).create(prog='dot', format='svg'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "encoder = Model(inputs = inputs, outputs = code)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "decoder_input = Input(shape = (code_dim,))\n",
    "decoder_output = auto_encoder.layers[-1]\n",
    "decoder = Model(inputs = decoder_input, outputs = decoder_output(decoder_input))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "auto_encoder.compile(optimizer='adam', loss='binary_crossentropy')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Training the model\n",
    "- Note that the features and targets are identical (i.e., **X_train**)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 1min 28s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x1eaead4dd68>"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "auto_encoder.fit(X_train, X_train, epochs = 1000, batch_size = 50, validation_data = (X_test, X_test), verbose = 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "encoded = encoder.predict(X_test)\n",
    "decoded = decoder.predict(encoded)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1eaeaa1d780>"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "plt.figure(figsize = (10,4))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkMAAADsCAYAAAB37KKJAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAADshJREFUeJzt3U2oXfW5BvB3m+BHtCYaEzHSDwU/EOp3bQU/AgF1UEkc\ndKID40gdGXEoanQggohROmhBidqORKIDcSClSarVgaQaEamImmApaqKJJmlINFl3cHuxd3DxPOee\nnZN93t9v/GT9197/vdZ+sg7sdzQMQwEAdHXMbJ8AAMBsUoYAgNaUIQCgNWUIAGhNGQIAWlOGAIDW\nlCEAoDVlCABoTRkCAFqbn4RHo1H0c9Xz5s2LTmbZsmVR/oQTTojy8+dHL7eqqg4cOBDlP/roo3iN\nxDAMo5k4TrqX4/azn/0syi9YsCBe44MPPojyhw4ditdIzNReVk3+fi5evDheY9u2bVH+yy+/jNdI\nTMq1mb7XP/7xj8d0Jt9Lr839+/eP6Uz+26TsZSr9Tv75z38er5Fel7t3747XCO0chmHJD4VGyTiO\ndGMXLVqUxGvt2rVR/uKLL47y6flU5Ru7atWqeI3EXL1In3nmmSif7n1V1fLly6P8uC/SuVyG0v28\n9dZb4zVuu+22KJ+eU2pSrs3Vq1dH+XXr1o3nRP5Dem2+88474zmRf5uUvUyl34Hp919V/vl66aWX\n4jVCW4ZhuPyHQv5MBgC0pgwBAK0pQwBAa8oQANCaMgQAtKYMAQCtKUMAQGvKEADQmjIEALSmDAEA\nreXDugLpfKL0Z7zT0RdHYAbKnJX+jHs6+mDcx6+y//8f6U/mp+NSbrrppihfdUR+xn8ipPfB9evX\nR/mtW7dG+fS+X5W/hnGP45ir0mtm4cKFY1/jaOHJEADQmjIEALSmDAEArSlDAEBryhAA0JoyBAC0\npgwBAK0pQwBAa8oQANCaMgQAtKYMAQCtHVWzydJ5M+n8o+nMpprODKy5KJ0dtnLlyii/efPmKL9u\n3booz/+WXjvpfp511llRftu2bVGe76V7mc4aS49vbtiRs2bNmih/7bXXjulMvjepnxdPhgCA1pQh\nAKA1ZQgAaE0ZAgBaU4YAgNaUIQCgNWUIAGhNGQIAWlOGAIDWlCEAoDVlCABobayzyVLp3JR09tl0\nZpOlM7nm6sys9L1OTWdvmL5Vq1ZF+WeffTbKmzV25Kxdu3as+XTW1EUXXRTlq6pWr14d/5u5KL0u\nH3/88Sj/4IMPRvkHHnggylflryE1rllmngwBAK0pQwBAa8oQANCaMgQAtKYMAQCtKUMAQGvKEADQ\nmjIEALSmDAEArSlDAEBryhAA0NpoGIaph0ejqYcrn+u1Zs2aKJ/O2JnOfK1nnnkmyi9fvjxeIzEM\nw2gmjpPuZfq6Nm7cGOW//vrrKD+dGXCbNm0aaz41U3tZle9nOgtq/fr1UX779u1RfjrzhtLrf1wz\njf7HbF2b4/bSSy9F+fS+XzX++2ZqtvYyfa9XrlwZ5Tdv3hzl03mh05Ge0zQ+K1uGYbj8h0KeDAEA\nrSlDAEBryhAA0JoyBAC0pgwBAK0pQwBAa8oQANCaMgQAtKYMAQCtKUMAQGvKEADQmjIEALQ2f5wH\n3717d5RPBy8eCdMZ7kpu4cKFUX7VqlXxGg888ECUv+2226J8OtR3NqVDaNPBq+l1kw5prsrf74sv\nvjheYy5K34d0GOhNN90U5avywctz4btlNhyJa+Duu++O8ulw2nHxZAgAaE0ZAgBaU4YAgNaUIQCg\nNWUIAGhNGQIAWlOGAIDWlCEAoDVlCABoTRkCAFpThgCA1kbDMEw9PBpNPTwNq1evjvLpfKXpzKdJ\n1xj3fKphGEYzcZxx72U6a+rxxx+P8un8m6r887Vo0aIon87jmqm9rBr/fqaf6yMxO2rbtm1RPp3L\nlB5/Uq7N9L2ezty4cVu+fHmUf+edd6L8bO1les8Z92f6k08+ifJVVaPRjN3WZsqWYRgu/6GQJ0MA\nQGvKEADQmjIEALSmDAEArSlDAEBryhAA0JoyBAC0pgwBAK0pQwBAa8oQANCaMgQAtDZ/tk/gP61a\ntSrKr1+/Pspv3bo1ylcdnXN5JsG6devGevzpzLJKpbPM5rL0vUj3Z9euXVG+qurrr7+O8um8tLkq\nndO1cOHCKL958+YoX5Xf++fqXqavK52dmc5P3L59e5SfZJ4MAQCtKUMAQGvKEADQmjIEALSmDAEA\nrSlDAEBryhAA0JoyBAC0pgwBAK0pQwBAa8oQANDaaBiGqYdHox1V1WdYydHnp8MwLJmJA9nLWTdj\ne1llP48Crs25w17OLVPaz6gMAQDMNf5MBgC0pgwBAK0pQwBAa8oQANCaMgQAtKYMAQCtKUMAQGvK\nEADQmjIEALSmDAEArSlDAEBryhAA0JoyBAC0pgwBAK0pQwBAa8oQANCaMgQAtKYMAQCtKUMAQGvK\nEADQ2vwkPBqNhjAfncyZZ54Z5U866aQo/+2330b5qqp9+/ZF+R07dkT5Q4cORflhGLI39f+Q7uU0\njh/lly5dGuWHIT/9nTt3RvnDhw/HayRmai+rxr+f8+bNi/LLli2L8tPZz88//zzKT+f6T8zWtTnu\n+2x6be7ZsyfKV1V99tlnY18jMSn32dTxxx8f5dPv2Kqq3bt3R/nvvvsuXiO0cxiGJT8UispQav78\n7PD33HNPlL/yyiujfHrzrKp68803o/zvf//7KL9r164oPynSi+7WW2+N8gcOHIjyVVVPP/10lN+7\nd2+8xlx18sknR/n0Wk7/U1BVtW7duij/j3/8I8pPp6DNhmOPPTbK33XXXWPNb968OcpXVT3yyCNR\nftOmTVF+Op+vSZAW4bPPPjvKX3311VG+qmrDhg1RPv1P6jSuy+1TCfkzGQDQmjIEALSmDAEArSlD\nAEBryhAA0JoyBAC0pgwBAK0pQwBAa8oQANDaWH+B+oorrojyN998c5TfuHFjlP/www+jfFXVxx9/\nHOUn5Vdrx+2qq66K8ukvFq9ZsybKV+W/iJ7+uusk7f0xx2T/D7rjjjui/HXXXRfl77333ihfVXXi\niSdG+XT/xz2+Y6b86le/ivJ33313lP/zn/8c5afzS/8rVqyI8m+88UaU379/f5SfFAsWLIjyv/nN\nb6L8wYMHo3xVfk7pvWhcvybuyRAA0JoyBAC0pgwBAK0pQwBAa8oQANCaMgQAtKYMAQCtKUMAQGvK\nEADQmjIEALSmDAEArY11NtlPfvKTKJ/OdvrrX/8a5T/44IMoX1X1z3/+M8rP1Rk4p5xySpR/7LHH\novyGDRui/CuvvBLlq8Y302YSXXDBBVH+4YcfjvJXX311lN+yZUuUr6o6/fTTo/yxxx4b5SdlNll6\n33zooYei/Msvvxzlf/nLX0b5qqrvvvsuyk9nZtZcdP7550f5tWvXRvkbbrghyldVLVmyJMp/9dVX\nUX7Pnj1Rfqo8GQIAWlOGAIDWlCEAoDVlCABoTRkCAFpThgCA1pQhAKA1ZQgAaE0ZAgBaU4YAgNaU\nIQCgtbHOJktniKSzhp588sko/9xzz0X5qqrnn38+yn/yySdR/sCBA1F+tpx77rlRfsGCBVH+hRde\niPKLFy+O8lVVu3btiv/NXHXZZZdF+XfffTfK79ixI8ovXbo0yldVzZ+f3b4m5VpLvfXWW1H+X//6\nV5Q/77zzovw111wT5auqfvvb30b5w4cPx2tMghNOOCHKr1ixIso/+uijUf5HP/pRlK+qOvvss6N8\nOjPwjTfeiPJT5ckQANCaMgQAtKYMAQCtKUMAQGvKEADQmjIEALSmDAEArSlDAEBryhAA0JoyBAC0\npgwBAK2NdTbZpk2bovz1118/nhP5t3T2WVXVjTfeGOXffvvtKL9v374oP1sOHToU5T///PMof+ed\nd0b5dPZVVdUrr7wS5Tdu3Bjl9+7dG+Vn0/bt26P8e++9F+VXrlwZ5U866aQoX1X16aefRvkXX3wx\nyu/cuTPKz5aDBw9G+f3790f5iy66KMovWrQoyldVnXzyyVE+neGVzmObLZdeemmUT6+z119/Pcqf\nc845Ub4qn0u5devWeI1x8GQIAGhNGQIAWlOGAIDWlCEAoDVlCABoTRkCAFpThgCA1pQhAKA1ZQgA\naE0ZAgBaU4YAgNaUIQCgtbEOak0HV/7lL3+J8vPmzYvyZ5xxRpSvqrruuuuifDpAcFKkg1eXLVsW\n5c8999wo/9RTT0X5qqorrrgiyp9yyilR/o9//OOUs4cPH46OPdM++uijKD8ajaL8aaedFuU//vjj\nKF+VD7Xctm1blP/Tn/405ewwDNGxZ1P6uU4H4n744YdRvqrquOOOi/LpMNBJGdQ6f372lfzqq69G\n+S+++CLKT2fo7ltvvRXlt2zZEq8xDp4MAQCtKUMAQGvKEADQmjIEALSmDAEArSlDAEBryhAA0Joy\nBAC0pgwBAK0pQwBAa8oQANDaWGeTpfOM0tlR6ayxK6+8MspXVe3YsSPKf/bZZ/EakyB9XU888USU\nX7FiRZRfvHhxlK+qWrp0aZRfsmRJlN+wYcOUs7M9KymdNfe3v/0tyl9yySVR/tChQ1G+Kp8DmM59\nmhTpXK/bb789yp933nlR/rXXXovyVVV79uyJ8n//+9+j/M6dO6P8bHn99dej/Pvvvx/lL7zwwih/\n2WWXRfmq/JwOHjwYrzEOngwBAK0pQwBAa8oQANCaMgQAtKYMAQCtKUMAQGvKEADQmjIEALSmDAEA\nrSlDAEBryhAA0NpYh/XMmzcvyt9yyy1R/sYbb4zyGzdujPJVVffff3+U37dvX7zGJPj222+j/O9+\n97sof+qpp0b5++67L8pXVb355ptRPn0Ne/fujfKzKZ0H9PTTT0f5J598Msr/+te/jvJVVX/4wx+i\n/Ntvvx3lh2GI8rMlneuWfk5/8YtfRPlvvvkmyldVvfjii1F+165d8RqTIN3LdOZaOmPyq6++ivJV\n+f4fLdeZJ0MAQGvKEADQmjIEALSmDAEArSlDAEBryhAA0JoyBAC0pgwBAK0pQwBAa8oQANCaMgQA\ntDZK5oKMRqMdVbV9fKfDD/jpMAxLZuJA9nLWzdheVtnPo4Brc+6wl3PLlPYzKkMAAHONP5MBAK0p\nQwBAa8oQANCaMgQAtKYMAQCtKUMAQGvKEADQmjIEALSmDAEArf0XCda0C6kPxVsAAAAASUVORK5C\nYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1eaeaa1d780>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "n = 5\n",
    "for i in range(n):\n",
    "    # visualizing test data instances\n",
    "    ax = plt.subplot(2, n, i+1)\n",
    "    plt.imshow(X_test[i].reshape(8,8))\n",
    "    plt.gray()\n",
    "    \n",
    "    ax.get_xaxis().set_visible(False)\n",
    "    ax.get_yaxis().set_visible(False)\n",
    "    \n",
    "    # visualizing encode-decoded test data instances\n",
    "    ax = plt.subplot(2, n, i+n+1)\n",
    "    plt.imshow(decoded[i].reshape(8,8))\n",
    "    plt.gray()\n",
    "    \n",
    "    ax.get_xaxis().set_visible(False)\n",
    "    ax.get_yaxis().set_visible(False)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deep Autoencoder\n",
    "- Autoencoders can be composed of several layers, i.e., deep autoencoder\n",
    "\n",
    "<br>\n",
    "\n",
    "<img src=\"https://deeplearning4j.org/img/deep_autoencoder.png\" style=\"width: 800px\"/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def encoder_decoder(code_dim = 10):\n",
    "    inputs = Input(shape = (X_train.shape[1],))\n",
    "    code = Dense(50, activation= 'relu')(inputs)\n",
    "    code = Dense(50, activation = 'relu')(code)\n",
    "    code = Dense(code_dim, activation = 'relu')(code)\n",
    "    \n",
    "    outputs = Dense(50, activation = 'relu')(code)\n",
    "    outputs = Dense(50, activation = 'relu')(outputs)\n",
    "    outputs = Dense(X_train.shape[1], activation = 'sigmoid')(outputs)\n",
    "    \n",
    "    auto_encoder = Model(inputs = inputs, outputs = outputs)\n",
    "    auto_encoder.compile(optimizer = 'adam', loss = 'binary_crossentropy')\n",
    "    \n",
    "    return auto_encoder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "auto_encoder = encoder_decoder()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 1min 51s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x1eaecd747b8>"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "auto_encoder.fit(X_train, X_train, epochs = 1000, batch_size = 50, validation_data = (X_test, X_test), verbose = 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "decoded = auto_encoder.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1eaecd74588>"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "plt.figure(figsize = (10,4))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkMAAADsCAYAAAB37KKJAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAADn1JREFUeJzt3VmolWXfBvB7qQ1KZFnaADlkUlCRYGREkCdBReWWICiC\n9KABOsii4aBICzoMBSEiCIumkyjDiKLCJKKCAg2jbHAgaOBtsIHK1J734P0+ejv40Gt/e7ld6//7\nHV/7uZ+97vU869rPhvXvdV3XAACqmjDeJwAAMJ6UIQCgNGUIAChNGQIASlOGAIDSlCEAoDRlCAAo\nTRkCAEpThgCA0iYl4V6vF31d9cSJE6OTOfnkk6P85MmTo/ykSdGv21prbffu3VH+iy++iNdIdF3X\nG4vjpHvZb7Nnz47yU6ZMidfYunVrlN+3b1+8RmKs9rK1wd/P4447Ll5jx44dUf7777+P10gMyrWZ\nvtannHJKn87kb+m1+fvvv/fpTP5jUPYylX4mn3322fEa6XW5a9eueI3Qd13XTd9fqJeM40g39phj\njknibeXKlVF+/vz5UT49n9byjR0ZGYnXSAzrRfr4449H+XTvW2tt0aJFUb7fF+kwl6F0P6+//vp4\njWXLlkX59JxSg3JtLl26NMqvXr26PyfyX9Jrc9OmTf05kf8xKHuZSj8D08+/1vL317p16+I1Qh90\nXXfu/kL+TQYAlKYMAQClKUMAQGnKEABQmjIEAJSmDAEApSlDAEBpyhAAUJoyBACUpgwBAKXlw7oC\n6Xyi9Gu809EXB2EGytBKv8Y9HX3Q7+O3Zv//P9KvzE/HpSxZsiTKt3ZQvsZ/IKT3wbVr10b5zZs3\nR/n0vt9a/jv0exzHsEqvmalTp/Z9jUOFJ0MAQGnKEABQmjIEAJSmDAEApSlDAEBpyhAAUJoyBACU\npgwBAKUpQwBAacoQAFCaMgQAlHZIzSZL582k849GM5tqNDOwhlE6O2zx4sVRfuPGjVF+9erVUZ5/\nSq+ddD/nzJkT5Xfs2BHl+Vu6l+mssfT45oYdPMuXL4/yF110UZ/O5G+D+n7xZAgAKE0ZAgBKU4YA\ngNKUIQCgNGUIAChNGQIASlOGAIDSlCEAoDRlCAAoTRkCAEpThgCA0vo6myyVzk1JZ5+NZjZZOpNr\nWGdmpa91ajR7w+iNjIxE+SeeeCLKmzV28KxcubKv+XTW1DnnnBPlW2tt6dKl8c8Mo/S6XLVqVZS/\n//77o/yKFSuifGv575Dq1ywzT4YAgNKUIQCgNGUIAChNGQIASlOGAIDSlCEAoDRlCAAoTRkCAEpT\nhgCA0pQhAKA0ZQgAKK3Xdd2Bh3u9Aw+3fK7X8uXLo3w6Y2c087Uef/zxKL9o0aJ4jUTXdb2xOE66\nl+nvtWHDhij/008/RfnRzIB78803+5pPjdVetpbvZzoLau3atVF+586dUX4084bS679fM43+13hd\nm/22bt26KJ/e91vr/30zNV57mb7WixcvjvIbN26M8um80NFIz2kU75UPuq47d38hT4YAgNKUIQCg\nNGUIAChNGQIASlOGAIDSlCEAoDRlCAAoTRkCAEpThgCA0pQhAKA0ZQgAKE0ZAgBKm9TPg+/atSvK\np4MXD4bRDHclN3Xq1Cg/MjISr7FixYoov2zZsiifDvUdT+kQ2nTwanrdpEOaW8tf7/nz58drDKP0\ndUiHgS5ZsiTKt5YPXh6Gz5bxcDCugdtuuy3Kp8Np+8WTIQCgNGUIAChNGQIASlOGAIDSlCEAoDRl\nCAAoTRkCAEpThgCA0pQhAKA0ZQgAKE0ZAgBK63Vdd+DhXu/Aw6OwdOnSKJ/OVxrNfJp0jX7Pp+q6\nrjcWx+n3XqazplatWhXl0/k3reXvr2OOOSbKp/O4xmovW+v/fqbv64MxO2rHjh1RPp3LlB5/UK7N\n9LUezdy4flu0aFGU37RpU5Qfr71M7zn9fk9v3749yrfWWq83Zre1sfJB13Xn7i/kyRAAUJoyBACU\npgwBAKUpQwBAacoQAFCaMgQAlKYMAQClKUMAQGnKEABQmjIEAJSmDAEApU0a7xP4byMjI1F+7dq1\nUX7z5s1RvrVDcy7PIFi9enVfjz+aWVapdJbZMEtfi3R/fvzxxyjfWms//fRTlE/npQ2rdE7X1KlT\no/zGjRujfGv5vX9Y9zL9vdLZmen8xJ07d0b5QebJEABQmjIEAJSmDAEApSlDAEBpyhAAUJoyBACU\npgwBAKUpQwBAacoQAFCaMgQAlKYMAQCl9bquO/Bwr/ev1lqdYSWHnlld100fiwPZy3E3ZnvZmv08\nBLg2h4e9HC4HtJ9RGQIAGDb+TQYAlKYMAQClKUMAQGnKEABQmjIEAJSmDAEApSlDAEBpyhAAUJoy\nBACUpgwBAKUpQwBAacoQAFCaMgQAlKYMAQClKUMAQGnKEABQmjIEAJSmDAEApSlDAEBpyhAAUNqk\nJNzr9bowH53MnDlzovzhhx/e1/xobN++Pcr/8ssvUb7ruuxF/T+kezmK40f5U089NcpPmTIlyrfW\n2scffxzl9+7dG6+RGKu9bO3Q28+TTjopyp944olRvrXW/vjjjyif7n/XZS/poFyb8+bNi/JHH310\nlP/555+jfGutbdu2Lcrv27cvXiMxKHuZSu+bc+fOjdf49NNPo/zu3bvjNULfdV03fX+hXnLBpxt7\n2GGHJfH2zDPPRPmZM2dG+VNOOSXKj8b1118f5V977bUoPygX6ZFHHhnln3766Si/YMGCKN9aa+ef\nf36U/+abb+I1EoNUho444ogof99990X5O+64I8q31tonn3wS5RcuXBjl07I1XtfmpEnR37Rt/fr1\nUf7iiy+O8hs2bIjyrbV29dVXR/kff/wxXiMxKPfZVHrffOGFF+I10vfL1q1b4zVCH3Rdd+7+Qv5N\nBgCUpgwBAKUpQwBAacoQAFCaMgQAlKYMAQClKUMAQGnKEABQmjIEAJSWfXVpaMaMGVH+kksuifIP\nPvhglH/llVeifGv5txZv2bIlXmMY3XnnnVF+ZGQkyj/wwANRvrXWvvvuu/hnhtWECdnfQY8++miU\nv+iii6L8vffeG+Vby8frHISv/R8XS5YsifLpffbzzz+P8unet9bas88+G+UvvfTSKJ+OVhkU6beP\nr1mzJsqnY3haa2369P1OvviHg/AN1AfEkyEAoDRlCAAoTRkCAEpThgCA0pQhAKA0ZQgAKE0ZAgBK\nU4YAgNKUIQCgNGUIAChNGQIASuvrbLITTzwxyn/77bdRfvPmzVH+m2++ifKttfbYY49F+T179sRr\nDIKJEydG+ZtvvjnKv/vuu1E+nUvXWmt79+6Nf2ZYXXDBBVF+8eLFUf68886L8p999lmUb214502l\n0pl76T3tlltuifKvv/56lG+ttd9++y3Kp7P19u3bF+UHxRVXXBHlzzzzzCg/mmtsUD8DPRkCAEpT\nhgCA0pQhAKA0ZQgAKE0ZAgBKU4YAgNKUIQCgNGUIAChNGQIASlOGAIDSlCEAoLRDajbZ3Llzo/wL\nL7wQ5f/8888o31przz//fJS/6aabovzu3buj/HiZN29elD/++OOj/K233hrl//rrryjPPy1atCjK\np7Pp0jmA6fFH8zODcq2lNm7cGOXfeeedKD9nzpwoP2vWrCjfWmt33313lB/WWWMzZsyI8k888USU\n//XXX6N8ej6ttTZ58uQov3Dhwij/3nvvRfkD5ckQAFCaMgQAlKYMAQClKUMAQGnKEABQmjIEAJSm\nDAEApSlDAEBpyhAAUJoyBACUpgwBAKX1dTZZOgPn9ttvj/JvvfVWlB/NbLKXXnopys+ePTvKb926\nNcqPl9NPPz3KH3744VH+nnvuifIXX3xxlG+ttTVr1kT5LVu2xGsMimOPPTbKH3XUUVH+ySefjPLp\n7LvWWvvoo4+i/LXXXhvl9+zZE+XHSzqnL/290utgwoT8b+xvv/02/plhtHbt2iif3mfTeX6jmRn4\n8ssvR/mPP/44yi9YsCDKHyhPhgCA0pQhAKA0ZQgAKE0ZAgBKU4YAgNKUIQCgNGUIAChNGQIASlOG\nAIDSlCEAoDRlCAAoTRkCAErr66DWH374IcqvWrWqT2fyH9OmTYt/Jh3uOijDHVPpUMzUcccdF+Uv\nvPDCeI1LLrkkyl955ZVRfvPmzVF+PD366KNR/uabb47yzz33XJQ/4YQTonxrrd10001R/pxzzony\n77//fpQfFKeddlqU//XXX6N8OkC7tdYeeeSRKL9y5coo//bbb0f58fLbb79F+V27dkX5SZOyj/z0\nfFpr7Y033ojyr776arxGP3gyBACUpgwBAKUpQwBAacoQAFCaMgQAlKYMAQClKUMAQGnKEABQmjIE\nAJSmDAEApSlDAEBpfZ1Nls5Bufzyy6P8u+++G+VvvPHGKN9aa19//XWU/+qrr+I1BsHOnTuj/MMP\nPxzl071/6qmnonxrrS1dujTKL1myJMoP0myyrVu3Rvlly5ZF+VmzZkX5DRs2RPnWWjvjjDOifHpO\ngzKbbOLEiVH+zjvvjPJffvlllE9f59ZaO+uss6J8en8ZFNdcc02UnzAhe55x2WWXRfk1a9ZE+dZa\nu+qqq6L8oTLP05MhAKA0ZQgAKE0ZAgBKU4YAgNKUIQCgNGUIAChNGQIASlOGAIDSlCEAoDRlCAAo\nTRkCAErr62yyXq8X5ZcvXx7lFyxYEOW3bNkS5Vtr7YYbbojyf/zxR7zGIEjnx6Tzj6ZNmxbl77rr\nrijfWmvbtm2L8s8880y8xrBat25dlH/ooYei/Pr166N8a6398ssvUf7FF1+M1xgEXddF+ZkzZ0b5\n2bNnR/l33nknyrfW2nXXXRflP/zww3iNQbB3796+Hn/fvn1R/vfff+/7GocKT4YAgNKUIQCgNGUI\nAChNGQIASlOGAIDSlCEAoDRlCAAoTRkCAEpThgCA0pQhAKA0ZQgAKK2XzLXp9Xr/aq3t7N/psB+z\nuq6bPhYHspfjbsz2sjX7eQhwbQ4PezlcDmg/ozIEADBs/JsMAChNGQIASlOGAIDSlCEAoDRlCAAo\nTRkCAEpThgCA0pQhAKA0ZQgAKO3fKPKGCTLhTZIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1eaecd74588>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "n = 5\n",
    "for i in range(n):\n",
    "    # visualizing test data instances\n",
    "    ax = plt.subplot(2, n, i+1)\n",
    "    plt.imshow(X_test[i].reshape(8,8))\n",
    "    plt.gray()\n",
    "    \n",
    "    ax.get_xaxis().set_visible(False)\n",
    "    ax.get_yaxis().set_visible(False)\n",
    "    \n",
    "    # visualizing encode-decoded test data instances\n",
    "    ax = plt.subplot(2, n, i+n+1)\n",
    "    plt.imshow(decoded[i].reshape(8,8))\n",
    "    plt.gray()\n",
    "    \n",
    "    ax.get_xaxis().set_visible(False)\n",
    "    ax.get_yaxis().set_visible(False)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One can see that deep autoencoder generates results closer to the original data"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
